Merging this PR will not alter performance

✅ 1057 untouched benchmarks
⏩ 1346 skipped benchmarks¹

_{Comparing Lychee-Technology:develop (8354deb) with develop (3029e9b)}

Almost there, thanks for taking the time to address the comments!

Hi @0ax1 , any other comments?

Almost there, thanks for taking the time to address the comments!

Hi @0ax1 , any other comments?

Currently swamped with other work but will get back to it when I find a free moment.

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?

I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?

I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

Could you please elaborate on what config refers to? If it’s S3 credentials, HTTP Proxy configuration, etc., I don’t believe we can read them from DuckDB. They are stored in Secrets Manager and the read API redacts sensitive information.

We probably can do the same as the httpfs extension does. Code link:

• 1
• 2

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?
I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

Could you please elaborate on what config refers to? If it’s S3 credentials, HTTP Proxy configuration, etc., I don’t believe we can read them from DuckDB. They are stored in Secrets Manager and the read API redacts sensitive information.

We probably can do the same as the httpfs extension does. Code link:

* [1](https://github.com/duckdb/duckdb-httpfs/blob/6afe39d5be688b032146baa97e49601b50c9456e/src/httpfs_extension.cpp#L73-L105)

* [2](https://github.com/duckdb/duckdb-httpfs/blob/main/src/create_secret_functions.cpp)

Let's handle this in a follow up to limit the scope of this PR.

Looks like this is materially worse for S3 performance:

[edit] looks like the coalescing configs are different from what ObjectStore uses. I think we just use the same values for now until we can properly investigate how best to tune these things.

Looks like this is materially worse for S3 performance: https://bench.vortex.dev/
[edit] looks like the coalescing configs are different from what ObjectStore uses. I think we just use the same values for now until we can properly investigate how best to tune these things.

Will follow up on that!

Looks like this is materially worse for S3 performance:

https://bench.vortex.dev/
[edit] looks like the coalescing configs are different from what ObjectStore uses. I think we just use the same values for now until we can properly investigate how best to tune these things.

Sorry, would you mind letting me know what are coalescing configs?

And I think it may be because the new version needs to install & load httpfs extension first. If you agree, I can make a PR to add the commands to the benchmark script.

Oh that's an interesting point. Definitely worth us pre-loading the extension, although it ships with core DuckDB so there wouldn't be any download.

Coalescing configs determine how to combine small individual requests to S3 into fewer larger requests to avoid the overhead of each request. We have made a PR to match them up to the old ones for one.

The odd thing for me is that the benchmarks didn't regress deterministically, they bounce up and down.

Sorry, would you mind letting me know what are coalescing configs?

And I think it may be because the new version needs to install & load httpfs extension first. If you agree, I can make a PR to add the commands to the benchmark script.

The install and load was addressed here: #6565. And the coalescing was fixed here: #6555. Must be sth else in the mix why the S3 perf regressed.

Oh that's an interesting point. Definitely worth us pre-loading the extension, although it ships with core DuckDB so there wouldn't be any download.

Coalescing configs determine how to combine small individual requests to S3 into fewer larger requests to avoid the overhead of each request. We have made a PR to match them up to the old ones for one.

The odd thing for me is that the benchmarks didn't regress deterministically, they bounce up and down.

Thanks @gatesn .

Based on my experience, even DuckDB doesn’t download the extensions, it still adds 2~4 seconds to my AWS Lambda (ARM 64-bit and 512MB memory) cold start time, just to install and load the parquet and httpfs extensions from my Lambda package. To address this issue, I created a project to build the static library of DuckDB with the parquet and httpfs extensions bundled in.